Papers with Structural Collapse-aware Optimization
A Few Bad Apples Spoil the Bunch: Preventing Global Entropy Collapse Driven by a Small Set of Tokens in LLM Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Reinforcement learning with verifiable rewards and Reinforced Learning from internal feedback fail to benefit from test-time compute due to entropy collapse and the resulting loss of reasoning diversity. |
| Approach: | They propose a strategy that assigns each generated token a redistribution score and applies selective KL regularization to only the top 5% of tokens under this score. |
| Outcome: | The proposed model improves on both RLVR and RLIF models on math reasoning benchmarks, showing that targeted entropy control at a vanishingly small subset of tokens is sufficient to sustain reasoning diversity and effective test-time scaling. |